1. Covid and the course
2. Course pages
3. Course overview
4. Introduction to SLV
5. Some examples
6. Data Wrangling
7. Wrap-up

Parts of this week’s slides may be based on materials from previous iterations of Data Analysis and Visualization courses. The authors of these materials include, but may not be limited to: Erik-Jan van Kesteren, Daniel Oberski and Peter van der Heijden.

When figures and other external sources are shown, the references are included when the origin is known.

With the exception of the first lecture, all lectures are on location. There are some rules by which we obide:

Covid related:

• You are required to wear a face-mask when moving. Please do so.
• You may remove your mask when seated in the lecture hall, if you want. You do not have to.
  
• Make use of the available space in the lecture rooms.
• If you have doubts about your health, please skip class and contact a medical professional.
• Please take other peoples wishes, needs, rights and freedom into consideration.
• We have a seperate online Q&A session every week to allow for more student-teacher interaction.

Procedure related:

• The first lecture will be recorded because of schedule clashes.

• The on-location lectures will not be recorded.

  • If you are ill, ask your classmates to cover for you.

• If you feel that you are stuck, and the wait for the Q&A session is too long: open a GitHub issue here.

  • You are most likely not the only one with that question. You are simply the bravest or the first.
  • Do not contact the teachers via private chat or e-mail.
  • use a reprex to detail your issue, when code is involved.

• If you expect that you are going to miss some part(s) of the course, please notify me via a private MS-Teams message.

You can find all materials at the following location:

All course materials should be submitted through a pull-request from your Fork of

The structure of your submissions should follow the corresponding repo’s README. To make it simple, I will add an example for the first of each submission type.

If you are unfamiliar with GitHub, forking and/or pull-request, please study this exercise from one of my other courses. There you can find video walkthroughs that detail the process.

All three have a PhD in statistics and a ton of experience in development, data analysis and visualization.

Each weak we have the following:

• 1 Lecture on Monday @ 9am in Ruppert D
• 1 Practical (not graded). Must be submitted to pass. Hand in the practical before the next lecture
• 1 online Q&A session.
• Course materials to study. See the corresponding week on the course page.

Twice we have:

• Group assignments
• The assignment is made in teams.
• Each assignment counts towards 25% of the total grade. Must be > 5.5 to pass.

Once we have:

• Individual exam
• BYOD: so charge and bring your laptop.
• 50% of total grade. Must be > 5.5 to pass.

We will make groups on Monday Sept 13!

1. What are statistical learning and visualization?
2. How does it connect to data analysis?
3. Why do we need the above?
4. What types of analyses and learning are there?

• Who will win the election?
• Is the climate changing?
• Why are women underrepresented in STEM degrees?
• What is the best way to prevent heart failure?
• Who is at risk of crushing debt?
• Is this matter undergoing a phase transition?
• What kind of topics are popular on Twitter?
• How familiar are incoming DAV students with several DAV topics?

• Description:
  What happened?
• Prediction:
  What will happen?
• Explanation:
  Why did/does something happen?
• Prescription:
  What should we do?

• Exploratory:
  mining for interesting patterns or results
• Confirmatory:
  Testing hypotheses

• Exploratory Data Analysis:
  Describing interesting patterns: use graphs, summaries, to understand subgroups, detect anomalies, understand the data
  Examples: boxplot, five-number summary, histograms, missing data plots, …

• Supervised learning:
  Regression: predict continuous labels from other values.
  Examples: linear regression, support vector machines, regression trees, … Classification: predict discrete labels from other values.
  Examples: logistic regression, discriminant analysis, classification trees, …

image source

How do you think that data analysis relates to:

• “Data analytics”?
• “Data modeling”?
• “Machine learning”?
• “Statistical learning”?
• “Statistics”?
• “Data science”?
• “Data mining”?
• “Knowledge discovery”?

People from different fields (such as statistics, computer science, information science, industry) have different goals and different standard approaches.

• We often use the same techniques.
• We just use different terms to highligh different aspects of so-called data analysis.
• All the terms on the previous slides are not exact synonyms.
• But according to most people they carry the same analytic intentions.

In this course we emphasize on drawing insights that help us understand the data.

Source: wikimedia commons and MIMP summerschool slide 28

Challenger space shuttle - 28 Jan 1986 - 7 deaths

Source: ISLR2, pp. 2

Ginsberg, J., Mohebbi, M., Patel, R. et al. Detecting influenza epidemics using search engine query data. Nature 457, 1012–1014 (2009)

Eder, Maciej & Rybicki, Jan & Kestemont, Mike. (2016). Stylometry with R: A Package for Computational Text Analysis. The R Journal. 8. 107-121. 10.32614/RJ-2016-007.

Mayor, J., Sanders, N., Classen, A. et al. Elevation alters ecosystem properties across temperate treelines globally. Nature 542, 91–95 (2017)

Image source

Where would you place each example in the table?

• Does each example fit in just one cell of the table?
• For each of these analyses, we can ask some common questions, such as:
  • “How well does the model fit the data?”
  • “How well does the model do on new, unseen, data?”
  • …
    

Can we think of other common questions?

Can we think of an example of a case where the model did not do well?

• When high risk decisions are at hand, it paramount to analyze the correct data.

• When thinking about important topics, such as whether to stay in school, it helps to know that more highly educated people tend to earn more, but also that there is no difference for top earners.

• Before John Snow, people thought “miasma” caused cholera and they fought it by airing out the house. It was not clear whether this helped or not, but people thought it must because “miasma” theory said so

• Election polls vary randomly from day to day. Before aggregating services like Peilingwijzer, newspapers would make huge news items based on noise from opinion polls.

• If we know flu is coming two weeks earlier than usual, that’s just enough time to buy shots for very weak people.

• If we know how ecosystems are affected by temperature change, we know how our forests will change in the coming 50-100 years due to climate change.

• Scholars fight over who wrote various songs (Wilhelmus), treatises (Caesar), plays (Shakespeare), etc., with shifting arguments. By counting words, we can sometimes identify the most likely author of a text, and we can explain exactly why we think that is the correct answer.

• Biologists have been constructing the tree of life based on appearance of the animal/plant. But sometimes the outward appearance corresponds by chance. DNA is a more precise method, because there is more of it, and because it is more directly linked to evolution than appearance. But there is so much of it that we need automated methods of reconstructing the tree.

The examples have in common that data analysis and the accompanying visualizations have yielded insights and solved problems that could not be solved without them.

• On some level, humans do nothing but analyze data;
• They may not do it consistently, understandibly, transparently, or correctly, however;
• DAV help us process more data, and can keep us honest;
• DAV can also exacerbate our biases when we are not careful.

Source: Mike Lee

Data wrangling is the process of transforming and mapping data from one “raw” data form into another format. - The process is often iterative - The goal is to add purpose and value to the data in order to maximize the downstream analytical gains

Source: R4DS

• Discovering: The first step of data wrangling is to gain a better understanding of the data: different data is worked and organized in different ways.
• Structuring:The next step is to organize the data. Raw data is typically unorganized and much of it may not be useful for the end product. This step is important for easier computation and analysis in the later steps.
• Cleaning: There are many different forms of cleaning data, for example one form of cleaning data is catching dates formatted in a different way and another form is removing outliers that will skew results and also formatting null values. This step is important in assuring the overall quality of the data.
• Enriching: At this step determine whether or not additional data would benefit the data set that could be easily added.
• Validating: This step is similar to structuring and cleaning. Use repetitive sequences of validation rules to assure data consistency as well as quality and security. An example of a validation rule is confirming the accuracy of fields via cross checking data.
• Publishing: Prepare the data set for use downstream, which could include use for users or software. Be sure to document any steps and logic during wrangling.

Source: Trifacta

• Study ISLR Chapter 1
• Study R4DS Chapters 1 and 9-16
• Make and hand in the practical before the next lecture

If you are unfamiliar with R, this crash course into scripting may be helpful